Classification by Attention: Scene Graph Classification with Prior Knowledge

A major challenge in scene graph classification is that the appearance of objects and relations can be significantly different from one image to another. Previous works have addressed this by relational reasoning over all an or incorporating prior knowledge into classification. Unlike previous works, we do not consider separate models for perception knowledge. Instead, take a multi-task learning approach introducing schema representations implementing as attention layer between image-based schemata. This allows emerge propagate within model. By enforcing model also represent prior, achieve strong inductive bias. We show our accurately generate commonsense iterative injection representations, top-down mechanism, leads higher performance. Additionally, fine-tuned on external given triples. When combined with self-supervised 1% annotated images only, gives more than 3% improvement object classification, 26% 36% predicate prediction accuracy.